UCM/UCS: Fail to create memtype cache if cannot patch Cuda driver API #7865

yosefe · 2022-01-22T21:28:59Z

Why

Many Cuda applications use static link, so it's not safe to assume that relocation-based memory hooks on Cuda runtime API can be enough. To be on the safe safe, fail to create the registration cache and (by default) fallback to pointer-query based memory detection.
Note: As a side effect, this will also disable RDMA registration cache when could not install Cuda hooks, since not knowing about a Cuda memory release can lead to stale memory keys in the cache and a data corruption.

How

Use Cuda Bistro hooks by default
UCX_MEMTYPE_CACHE can by yes/no/try (default: try). If set to yes and it could not be created - print a warning.
Memtype cache fails to create if could not set memtype hooks

The logic is:

UCX_MEM_CUDA_HOOK_MODE=bistro (default):
- If bistro hooks on driver API fail - registration cache and memory type cache are not created.
  If also UCX_MEMTYPE_CACHE=y (default is try) - a warning is printed.
UCX_MEM_CUDA_HOOK_MODE=reloc:
This method sets reloc hooks on both driver and runtime APIs.
If either driver or runtime API reloc hooks fail (it's not expected) - memtype cache and rcache are disabled.
This method is safe only if the application is dynamically linked to cuda runtime API (or using the driver API directly), so it's not the default.

Debugging

Set "UCX_MEM_LOG_LEVEL=diag" to get more info if failed to install memory hooks.

Related to #7791 (comment)
cc @Akshay-Venkatesh @pentschev

shamisp · 2022-01-23T16:50:53Z

@yosefe is this only for static or this will impact dynamic linked cuda applications as well ?

yosefe · 2022-01-23T17:29:15Z

@yosefe is this only for static or this will impact dynamic linked cuda applications as well ?

it will affect both, to use only driver-api hooks instead of runtime/driver.

shamisp · 2022-01-23T19:03:38Z

@yosefe in the past you have been avoiding this because of high cost (system call). Does it mean that performance will go down ?

yosefe · 2022-01-23T19:09:04Z

@yosefe in the past you have been avoiding this because of high cost (system call). Does it mean that performance will go down ?

Do you mean memory type detection?
We still use memtype cache for detection (if possible); it's just that instead of hooking both driver and runtime APIs for it, we actually need to hook only the driver API.
The latest observation is that we really want to ensure the driver API hooks are installed [since any app can be static link potentially], so there is no point of hooking the upper-layer runtime APIs.

shamisp · 2022-01-23T19:20:11Z

@yosefe got it. thanks

yosefe · 2022-01-26T18:23:49Z

@Akshay-Venkatesh @bureddy can you pls take a look?

yosefe · 2022-01-27T15:45:41Z

per offline discussion with @bureddy and @Akshay-Venkatesh : will keep reloc hooks for Cuda runtime API as a disabled option, for applications that use dynamic link and fail to patch driver API for some reason.

yosefe · 2022-02-03T15:59:24Z

@Akshay-Venkatesh @bureddy can you pls take a look?

Akshay-Venkatesh · 2022-02-03T16:59:30Z

@Akshay-Venkatesh @bureddy can you pls take a look?

Sorry for the delay @yosefe Taking a look now.

Akshay-Venkatesh

@yosefe Probably missed this is in the diff but
~~1. Can you point to how driver hooks installation failure prevents memtype_cache from being created?~~

Is it the case that memtype_cache creation calls ucm_cudamem_install (which doesn't return UCS_OK if driver hooks failed) and this leads to disabling memtype cache when bistro and reloc methods fail on driver functions?
~~2. Is it the case that memtype_cache being absent prevents rcache instance from being created and avoids potential data corruption?~~

Akshay-Venkatesh · 2022-02-03T17:23:07Z

src/ucm/cuda/cudamem.c

+        goto out_unlock;
+    }
+
+    status = ucm_cuda_install_hooks(ucm_cuda_runtime_funcs, "runtime",


Do we still need to attempt installation of runtime hooks here? If we're not going to create memtype cache if driver hooks failed, can we not skip this step?

if driver bistro hooks failed, we do skip the other steps:

status = ucm_cuda_install_hooks(ucm_cuda_driver_funcs, "driver", UCM_MMAP_HOOK_BISTRO, &driver_api_hooks); if (status != UCS_OK) { **goto out_unlock;** }

I mean can we not remove lines 282-285?

these lines are needed in case the user would like to set UCX_MEM_CUDA_HOOK_MODE=reloc manually

bureddy · 2022-02-03T17:59:51Z

src/ucs/memory/memtype_cache.c

+            ucs_memtype_cache_failed = 1;
+            if (ucs_global_opts.enable_memtype_cache == UCS_YES) {
+                ucs_warn("failed to create memtype cache: %s",
+                         ucs_status_string(status));


should it be a hard error if the user is forced to enable memtype cache?

i don't think it should be a fatal error since we can still continue the program without memtype cache

Akshay-Venkatesh · 2022-02-03T18:59:18Z

@Akshay-Venkatesh @bureddy can you pls take a look?

@yosefe If you don't mind, can you include in PR description, the current conditions under which

bistro hooks fail with driver API
reloc hooks fail with driver API

yosefe · 2022-02-03T19:12:26Z

@yosefe If you don't mind, can you include in PR description, the current conditions under which

bistro hooks fail with driver API

reloc hooks fail with driver API

@Akshay-Venkatesh I've updated PR desc. Hope it answers the previous questions as well.

- Reloc hooks are optional but are not the default. Default is bistro hooks on Cuda driver API. - If UCM fails to install the configured hooks, do not create the memory type cache. - If failed to create memtype cache once, don't try again. - Use getauxv() API if possible instead of reading /proc/self/auxv directly - fixes permissions errors on some systems. - Enable Cuda bistro hooks also with valgrind, since it doesn't affect heap memory allocations. - Fix error message in tests.

yosefe force-pushed the topic/ucm-ucs-fail-to-create-memtype-cache branch from 7c67e7b to 537eb4b Compare January 22, 2022 21:42

yosefe added the Bugfix label Jan 22, 2022

yosefe force-pushed the topic/ucm-ucs-fail-to-create-memtype-cache branch from 537eb4b to 8e1b3be Compare January 23, 2022 10:29

Akshay-Venkatesh reviewed Feb 3, 2022

View reviewed changes

Akshay-Venkatesh approved these changes Feb 3, 2022

View reviewed changes

bureddy reviewed Feb 3, 2022

View reviewed changes

bureddy approved these changes Feb 3, 2022

View reviewed changes

yosefe force-pushed the topic/ucm-ucs-fail-to-create-memtype-cache branch from 2fd5f72 to 119a81b Compare February 3, 2022 20:35

yosefe merged commit 5a57268 into openucx:master Feb 5, 2022

yosefe deleted the topic/ucm-ucs-fail-to-create-memtype-cache branch February 5, 2022 13:25

yosefe mentioned this pull request Feb 5, 2022

Memtype cache fixes - v1.12.x #7909

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

UCM/UCS: Fail to create memtype cache if cannot patch Cuda driver API #7865

UCM/UCS: Fail to create memtype cache if cannot patch Cuda driver API #7865

yosefe commented Jan 22, 2022 •

edited

Loading

shamisp commented Jan 23, 2022

yosefe commented Jan 23, 2022

shamisp commented Jan 23, 2022

yosefe commented Jan 23, 2022

shamisp commented Jan 23, 2022

yosefe commented Jan 26, 2022

yosefe commented Jan 27, 2022

yosefe commented Feb 3, 2022

Akshay-Venkatesh commented Feb 3, 2022

Akshay-Venkatesh left a comment •

edited

Loading

Akshay-Venkatesh Feb 3, 2022

yosefe Feb 3, 2022

Akshay-Venkatesh Feb 3, 2022

yosefe Feb 3, 2022

bureddy Feb 3, 2022 •

edited

Loading

yosefe Feb 3, 2022

Akshay-Venkatesh commented Feb 3, 2022

yosefe commented Feb 3, 2022

UCM/UCS: Fail to create memtype cache if cannot patch Cuda driver API #7865

UCM/UCS: Fail to create memtype cache if cannot patch Cuda driver API #7865

Conversation

yosefe commented Jan 22, 2022 • edited Loading

Why

How

Debugging

shamisp commented Jan 23, 2022

yosefe commented Jan 23, 2022

shamisp commented Jan 23, 2022

yosefe commented Jan 23, 2022

shamisp commented Jan 23, 2022

yosefe commented Jan 26, 2022

yosefe commented Jan 27, 2022

yosefe commented Feb 3, 2022

Akshay-Venkatesh commented Feb 3, 2022

Akshay-Venkatesh left a comment • edited Loading

Choose a reason for hiding this comment

Akshay-Venkatesh Feb 3, 2022

Choose a reason for hiding this comment

yosefe Feb 3, 2022

Choose a reason for hiding this comment

Akshay-Venkatesh Feb 3, 2022

Choose a reason for hiding this comment

yosefe Feb 3, 2022

Choose a reason for hiding this comment

bureddy Feb 3, 2022 • edited Loading

Choose a reason for hiding this comment

yosefe Feb 3, 2022

Choose a reason for hiding this comment

Akshay-Venkatesh commented Feb 3, 2022

yosefe commented Feb 3, 2022

yosefe commented Jan 22, 2022 •

edited

Loading

Akshay-Venkatesh left a comment •

edited

Loading

bureddy Feb 3, 2022 •

edited

Loading